Intelligently interactive profiling system and method

ABSTRACT

One aspect of the invention is a method for identifying at least one property of data. An example of the method includes receiving data, and making assessments regarding the data. The method also includes applying at least one behavioral operator, and outputting results. The method further comprises receiving feedback concerning system performance. Additionally, the method includes adjusting at least one parameter based on the feedback received concerning system performance, wherein the at least one parameter is a parameter of a machine learning method.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 60/410,905, filed Sep. 13, 2002, titled, “AnIntelligently Interactive Profiling System”, which is incorporatedherein by this reference.

BACKGROUND

1. Technical Field

The present invention relates to identifying at least one property ofdata. More particularly, the invention concerns an intelligentlyinteractive system for identifying at least one property of data.

2. Description of Related Art

It is frequently useful to profile data. For example, data may beprofiled to determine the expected risk of fraud in a credit cardtransaction, or the risk of terrorism that a freight shipment poses, orthe risk that a patient has a serious medical condition. Data profilingcan also be applied to ascertain the chances that a viewer will enjoy amovie, the chances that a person will be compatible with another personin a dating service database, or the chances that a stock will go up ordown as the result of set of economic conditions.

Known methods of profiling data may involve applying behavioral rulesprescribed by human experts to the data. As an example, a behavioralrule could assign a high level of risk of fraud to a credit cardtransaction, if the credit card used for the transaction has beenreported lost. As another example, a behavioral rule could assign a highlevel of terrorism risk to a shipping container, if a high level ofradioactivity is measured outside the container.

One shortcoming of using only behavioral rules prescribed by humanexperts for profiling data is that the experts may have insufficientknowledge to prescribe rules. Other shortcomings of using only rulesprescribed by human experts are that the experts may erroneouslyprescribe incorrect rules, or may prescribe conflicting rules. Anothershortcoming is that humans typically cannot quickly develop rules, andare slow to react when there is a need to change rules. Yet anothershortcoming is that over time, the number of rules prescribed by humanexperts may grow very large and may require a long time to process,which could result in the profiling being too slow for manyapplications. For example, a method for profiling data to determine therisk of fraud for a credit card transaction must be able to be completedwithin several seconds in order to be useful for many applications.Another shortcoming of using only rules prescribed by human experts isthat some of the rules may be difficult or impossible to implement.Existing methods for profiling data have additional shortcomings, suchas not having an automatic feedback loop for improving the rulesprescribed by human experts, and not being reactive or proactive to theuser.

Existing methods for profiling data that utilize machine learning in theform of neural networks merely function as black boxes that produce anoutput, and also are not reactive or proactive to the user. The lack ofuser feedback in these methods limits the accuracy of the results, andlimits the capability of these methods to adapt to changed circumstancesor to correct errors. Further, existing methods that utilize machinelearning rely excessively on supervised learning, which may limit theaccuracy and usefulness of the results in cases where feedback islimited or nonexistent.

In summary, existing methods for profiling data are inadequate for manyapplications.

SUMMARY

One aspect of the present invention concerns a method for identifying atleast one property of data. An example of the method includes theoperations of receiving data, and making assessments regarding the data.The method also includes applying at least one behavioral operator, andoutputting results. The method further comprises receiving feedbackconcerning system performance. Additionally, the method includesadjusting at least one parameter based on the feedback receivedconcerning system performance, wherein the at least one parameter is aparameter of a machine learning method.

Other aspects of the invention are described in the sections below, andinclude, for example, a profiling system, and a signal bearing mediumtangibly embodying a program of machine-readable instructions executableby a digital processing apparatus to perform a method for identifying atleast one property of data.

The invention provides a number of advantages. For example, someexamples of the invention advantageously adjust at least one parameterof a machine learning method, based on feedback received from a user.The invention also provides a number of other advantages and benefits,which should be apparent from the following description.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of the hardware components andinterconnections of a system for identifying at least one property ofdata, in accordance with an example of the invention.

FIG. 2 is an example of a signal-bearing medium in accordance an exampleof the invention.

FIG. 3 is a block diagram illustrating interactions between functionalelements of a system for identifying at least one property of data, andinteractions between the system and a user, and data sources, inaccordance an example of the invention.

FIGS. 4A-C are a flowchart of an operational sequence for identifying atleast one property of data in accordance with an example of theinvention.

FIG. 5 is a depiction of a display of an output showing a plurality ofmembership functions and an indicator in accordance with an example ofthe invention.

DETAILED DESCRIPTION

The nature, objectives, and advantages of the invention will become moreapparent to those skilled in the art after considering the followingdetailed description in connection with the accompanying drawings.

I. Hardware Components and Interconnections

One aspect of the invention is a system for identifying at least oneproperty of data. The at least one property of the data may beassociated with profiling the data. As an example, the system may beembodied by the hardware components and interconnections of thecomputing system shown in FIG. 1, which will be referred to as theprofiling system 100.

The profiling system 100 includes a processor 104 coupled to a storage106. The storage 106 includes a memory 108 and a nonvolatile memory 110.As an example, the memory 108 may be RAM. The nonvolatile memory 110 maycomprise, for example, one or more magnetic data storage disks such as ahard drive, an optical drive, a tape drive, or any other suitablestorage device. The storage 106 may store programming instructionsexecuted by the processor 104. The profiling system 100 also includes anoutput 112, and may also include a display 114 that is coupled to theoutput 112. A keyboard 116 may also be included in the profiling system100. The profiling system 100 also includes an input/output 118, such asa line, bus, cable, electromagnetic link, or other means for theprofiling system 100 to send or receive data from external to theprofiling system 100. As an example, data may be inputted to theprofiling system 100 via the input/output 118.

The profiling system 100 may be implemented by any suitable computingapparatus, such as, for example, a super computer, a mainframe computer,a computer workstation, a personal computer, a cluster of computingdevices, or a grid of computing devices connected over a LAN or WAN.Classified or proprietary data should be stored on a secure machine, andin the case of a cluster or grid, should be stored on a secure network.In one example, the profiling system 100 is a personal computer with anIntel processor running the Windows operating system, having the maximumavailable computing power and data access rates, and having thecapability to back up data.

The profiling system may be implemented in a machine of differentconstruction than the profiling system 100 described above, withoutdeparting from the scope of the invention. As an example, the memory 108or the nonvolatile memory 110 may be eliminated, or the storage 106could be provided on-board the processor 104, or the storage 106 couldbe provided remotely from the processor 104.

II. Operation

In addition to the various hardware embodiments described above, anotheraspect of the invention concerns a method for identifying at least oneproperty of data.

A. Signal-Bearing Media

In the context of FIG. 1, the method may be implemented, for example, byoperating the profiling system 100 to execute a sequence ofmachine-readable instructions, which can also be referred to as code.These instructions may reside in various types of signal-bearing media.In this respect, one aspect of the present invention concerns aprogrammed product, comprising a signal-bearing medium or signal-bearingmedia tangibly embodying a program of machine-readable instructionsexecutable by a digital processing apparatus to perform a method foridentifying at least one property of data.

This signal-bearing medium may comprise, for example, the memory 108 orthe nonvolatile memory 10. Alternatively, the instructions may beembodied in a signal-bearing medium such as the optical data storagedisc 200 shown in FIG. 2. The optical disc could be any type of signalbearing disc, for example, a CD-ROM, CD-R, CD-RW, WORM, DVD-R, DVD+R,DVD-RW, or DVD+RW. Whether contained in the profiling system 100 orelsewhere, the instructions may be stored on any of a variety ofmachine-readable data storage mediums or media, which may include, forexample, direct access storage (such as a conventional “hard drive”, aRAID array, or a RAMAC), a magnetic data storage diskette (such as afloppy disk), magnetic tape, digital optical tape, RAM, ROM, EPROM,EEPROM, flash memory, magneto-optical storage, paper punch cards, or anyother suitable signal-bearing media, including transmission media suchas digital and/or analog communications links, which may be electrical,optical, and/or wireless. As an example, the machine-readableinstructions may comprise software object code, compiled from a languagesuch as “C++”.

B. Operational Modules of Profiling System

FIG. 3 is a block diagram illustrating interactions between functionalelements of a profiling system 300, and interactions between theprofiling system 300 and a user 302 and data sources 306, 308, 310. Theprofiling system may, for example, perform a method for identifying atleast one property of data. The profiling system 300 may be called anintelligently interactive profiling system.

As an example, the profiling system 300 may be implemented by a softwareprogram that may run on the profiling system 100 described above. Theprogram may be embodied in signal-bearing media, as discussed above. Theprofiling system 300 includes an interface/control module 304 that mayreceive input from the user 302 and present output to the user 302. Theinterface/control module 304 also may receive data from one or more datasources including a commercial data source 306, a government data source308, and/or other data sources 310. The interface/control module 304 iscoupled to a data integrity module 312 which examines the integrity ofthe data received by the interface/control module 304. Theinterface/control module 304 is also coupled to a behavioral operatorsmodule 314, which generates and evaluates behavioral operators used bythe profiling system 300. The behavioral operators module 314 may befurther configured to adjust one or more behavioral operators, and toadd new behavioral operators, based on feedback received regarding theoutputted results. An anomaly detection module 316, for detectinganomalies in the data, is also coupled to the interface/control module304. The interface/control module 304 is also coupled to a machinelearning module 318, which performs machine learning, such as supervisedand/or unsupervised learning, which may be performed during the processof identifying at least one property of data (which in some examples maybe a process for interactive profiling). The interface/control module304 may receive feedback regarding outputted results from the user 302.The interface/control module 304 may also receive feedback concerningsystem performance from the user 302, and may adjust parameters based onthe feedback received concerning system performance. Theinterface/control module 304 may also proactively generate at least onesuggestion and output the at least one suggestion to the user 302, andsolicit feedback from the user 302 concerning the generated suggestions.The interface/control module 304 may also receive feedback concerningthe generated suggestions, and interpret the feedback receivedconcerning the generated suggestions. The operation of the profilingsystem 300 is discussed further below.

C. Overall Sequence of Operation

An example of the method aspect of the invention is a method foridentifying at least one property of an item(s), individual(s), or otherdata. The properties identified may be desired properties and/orundesired properties. In some embodiments, the method may be referred toas a method for profiling multiple items and/or individuals andoutputting results regarding the items and/or individuals, in anintuitive, user-friendly manner. A user may first identify a problem,which may be a recognized deficiency between a current state and adesired state, and a performance measure may be constructed to capturethe preferences of outcomes. In general, the user 302 may query theprofiling system 100 with respect to a particular item or individual, orpossibly multiple items and/or individuals. The user 302 may access theprofiling system 100 via the keyboard 116, or via an external terminal,remote dial-in, over the Internet, or via other means of access.

The profiling system l00 may respond by identifying one or morecharacteristics of the item or individual, such as the terrorism riskposed by the item (such as a shipping container) or the chances that theuser 302 would enjoy dating the individual. The following are examplesof properties of items, individuals, or other data, that may beidentified by the method aspect of the invention: the (terrorism) riskpresented by a shipment; the (terrorism) risk presented by a shippingcontainer; the risk that an individual is a terrorist; the risk of fraudassociated with a credit transaction; the risk that cancer (or thatanother disease) is present; the chances that a person will enjoy datinganother person; the chances that a person will enjoy a particular movie;etc. Additional examples include: evaluating (assessing) a restaurant todetermine whether or not someone would like to eat there; evaluatingproperty (e.g., a house, a car) to determine whether or not someonewould like to buy or sell it; evaluating customer data to determinewhich customers would be better selections for marketing alternativeproducts; evaluating applications for a job or entrance to a universityfor suitability; evaluating records of financial filings to determinewhich records may contain fraud or errors; evaluating candidates fordrugs to determine which may be particularly appropriate for addressinga chosen disease; evaluating alternative investments to determine whichare appropriate or inappropriate for an investor; evaluating sportingequipment to determine which among a range of possible choices is bestfor the player; evaluating alternative vacation destinations todetermine which the user would be most or least likely to enjoy;evaluating employee performance to determine if a promotion isappropriate; and evaluating data pertinent to the health of equipmentand predicting the onset of failures.

An example of the method aspect of the present invention is illustratedin FIGS. 4A-C, which show a sequence 400 for a method for identifying atleast one property of an item(s), individual(s), or other data. For easeof explanation, but without any intended limitation, the example ofFIGS. 4A-C is described in the context of the profiling system 100 andthe profiling system 300 described above. In one example, the operationsof the sequence 400 are performed in the order that they appear in FIGS.4A-C. However, it will be apparent to persons skilled in the art thatthe order of performance of the operations of the sequence 400 in manycases may be different than the order in FIGS. 4A-C.

Referring to FIG. 4A, the sequence 400, which may be performed by theprofiling system 100, may begin with operation 404, which comprisesreceiving data, which may also be called gathering the data. The datamay be received from the commercial data source 306, the government datasource 308, or from other data sources 310. The received data may be,for example, a historical set of credit card transactions, a shipmenthistory of a shipping company, which possibly could be obtained fromcommercial transaction data, and could also be local, state, federal orforeign government data or data from the United Nations. Data could alsobe obtained by making direct measurements. The profiling system 100 mayinput data once, repeatedly, or continuously. To receive the data, theprofiling system 100 may access databases such as the commercial datasource 306, the government data source 308, and/or other data sources310, which nominally may contain data regarding the item(s) and/orindividual(s). In a shipping container profiling embodiment, theinputted data may include, for example, facts involving priortransactions, transportation of goods, responsible parties, criminalrecords, known associates, and other data that may be pertinent toevaluating the item(s) and/or individual(s) with respect to the purposeof the user 302. In some examples the profiling system may interrogatethe available databases with respect to diverse methods of evaluatingthe profile desired. The inputted data may be stored in the storage 106.However, the inputted data does not have to be stored in a singlestorage device or location.

The sequence 400 may also include operation 406, which comprises makingassessments regarding features and/or the data. The operation of makingassessments regarding features and the data in many instances mayinclude checking the integrity of the data. The data are typicallyassessed with respect to issues such as:

-   completeness: (determining if any data are missing, and how many    data are missing), reliability (determining if the data were    collected at the proper time, and determining if the correct data    are being collected);-   precision: (determining how precise the data are, and determining if    measurements are being made with sufficient resolution for example,    determining if measurements to the nearest meter are being taken    when measurements to the nearest centimeter are needed);-   accuracy (determining how accurate the data are, and determining if    the data are often in error, and determining why the data are in    error for example, determining if there is noise that is inherent to    the measurements, and determining if there is noise in the system,    or in the sensors, or both, and determining how much human error is    involved).

Assessing the data in the shipping container profiling embodiment mayinclude, for example, verifying facts regarding the item(s) and/orindividual(s) in question, such as matching a vehicle identificationnumber with a particular make of car, or a passport number with aparticular individual. As another example of checking data integrity, inan embodiment for determining which movies a user 302 may like,information regarding how much money a particular movie made could begathered from at least two sources, and the information could becompared to check for consistency.

Assessing features typically involves discovering features, as well asdetermining the utility of the features. Discovering features is aprocess of noting repeated patterns over examples, which could betime-dependent or static. For example, one feature of the moon is thatthe bright side always points to the sun. The utility of a feature comesin how it is used to accomplish some task. For example, once it is knownthat the bright side of the moon always faces the sun, a relationshipbetween the moon and the sun can be imputed to try to understand thedynamics of how that feature could arise. Features may be mined fromdata by looking for patterns that are either repeated simply as patternsthemselves, or as patterns associated with events, such as a seasonalvariation in sales for clothing.

The interface may allow the user 302 to make adjustments to theprofiling system 100 to incorporate the user's 302 knowledge. Thus, thesequence 400 may also include operation 408, which comprises receivinginput from the user 302 concerning the user's knowledge. For example,the user 302 may know that a certain credit card has been stolen butthis fact has not yet been reported to commercial data warehouses. Basedon this knowledge, the user 302 could enter a rule identifying thestatus of the credit card number with an appropriate level of profilingassessment. (Profiling assessment is different than assessing the datafor data integrity.) As another example, the user 302 could haveknowledge of the perceived risk associated with various countries ororganizations in the world, which could serve to influence the perceivedrisk associated with shipments of goods from those countries, or goodsthat have been transported through those countries.

The sequence 400 may also include operation 410, which comprisesapplying at least one behavioral operator. Behavioral operators mayinclude behavioral rules and/or suitable mathematical constructs (suchas a neural network). The behavioral operators represent conditionsand/or behavior that are of interest to the user 302, and are usuallybased on features of the data.

Applying behavioral rules involves developing explicit rules pertainingto features, and to associated patterns of behavior. The behavioralrules are typically conclusions or actions to take (behavior=stimulus-response pair), based on conditions detected. For example, ifsomeone smiles and then shakes your hand, and then smiles, and thenshakes your hand, you might develop the rule IF [Person Smiles] THEN[extend hand]. This is a behavior rule, applied to features detectedbased on sensed data. Another example of a behavioral rule is: If dataindicate that a shipping container was sent from a known criminal, thenincrease the perceived risk associated with the container. The followingis another example of a behavioral rule: If data indicate that apotential immigrant has traveled to countries designated to be ofconcern to the government, then increase the perceived risk associatedwith the individual. Yet another example of a behavioral rule is: Ifdata indicate that a carrier has a record of significant violations oflaws in prior shipments, then increase the perceived risk associatedwith the carrier and/or item. One example of a behavioral operator,other than a behavioral rule, is a neural network, which in a gamingexample, may “profile” a checkerboard based on input features todetermine the favorability of the particular position of checkers on theboard.

The sequence 400 may also include operation 412, which comprisesoutputting results, so that the results can be displayed for the user302 (which can also be called reporting the results). The results arethe results of applying at least one behavioral operator to the data.The behavioral rules may also be outputted. The behavioral rules may bereported according to user preferences, for example, with pros and conssorted separately. The results of the profiling system 100 may bepresented to the user 302 with a graphical-user interface. The resultspresented on the interface may indicate overall profiles, level of riskassessed (possibly with associated color coding representing variousconditions), as well as a report on the rationale of the profilingsystem's evaluation so that the user 302 can understand why the systemhas reached the decision that it reached and what information might berequired to make an improved evaluation. As an example, the operation ofreporting results may comprise outputting information which isconfigured to display a plurality of membership functions, and anindicator showing the relationship between the results and themembership functions, as is shown in FIG. 5.

FIG. 5 is a depiction of a display 500 of an output showing a pluralityof membership functions and an indicator 502. In the display 500 of theoutput of the profiling system 100, membership functions are displayedas triangles, which, for example, may depict a particular shipment'sdegree of membership in a linguistic risk categorization. In thisembodiment, the display 500 can be called a risk assessment display. Therisk designations are compatible with fuzzy logic (which is a form ofapproximate reasoning).

FIG. 5 shows how five risk levels may be characterized by membershipfunctions. For example, a record that scored as indicated by the small,inverted arrow is a member of both the elevated risk level (indicated bythe large upright triangle centered under the number 0.5, and whichcould be colored yellow) and the overlapping high risk level (indicatedby the large, upright triangle centered under the number 0.66, and whichcould be colored orange). Thus, the score indicated by the smallinverted arrow is well into the elevated level, while it also registerstoward the low-end of the high level. The display 500 also includes alow risk level (indicated by the horizontal line and the downwardsloping line that intersect under the number 0.16, and which could becolored green), and also includes a guarded risk level (indicated by thelarge upright triangle under the number 0.33, and which could be coloredblue), and also includes a severe risk level (indicated by upwardsloping line and the horizontal line that intersect under the number0.87, and which could be colored red).

As an example, linguistic descriptions of risk levels may be defined interms of their respective lower edges as follows: 0.000 may mark thelower edge of the low risk level; 0.333 may mark the lower edge of theguarded risk level, 0.500 may mark the lower edge of the elevated risklevel, 0.667 may mark the lower edge of the high risk level, and 0.875may mark the lower edge of the severe risk level. In the illustratedexample, the center of the severe category is shifted slightly to theright (from 0.833 to 0.875). As shown in FIG. 5, the defining values forthe high, elevated, guarded, and low risk levels are centered at thevertexes of their respective triangles (although the membership functionfor low risk could be a trapezoidal function, symmetric to that of thesevere risk category). A linguistic interpretation of the risk, such aslow, guarded, elevated, high, or severe, may also be shown in thedisplayed output.

Linguistic descriptions of risk levels may also be defined in terms oftheir respective points of intersection between membership functions,indicating equivalent membership in a linguistic description above andbelow the associated numeric level of assessed risk.

Referring again to FIG. 4A, the sequence 400 may also include operation414, which comprises receiving feedback from the user 302 regarding theoutputted results. As an example, the user 302 may adjust the profilingsystem's 100 response by modifying the profiling rules in terms of theperceived risk or assessment. For example, a user 302 may choose to erron the side of caution and indicate higher risks in general operations.Another user may be less risk averse and choose to err on the side ofindicating a lower risk for certain items and/or individuals who mightotherwise have been profiled with some evident concern.

The sequence 400 may also include operation 416, which comprisesadjusting at least one of the behavioral operators based on the feedbackreceived regarding the outputted results. The user 302 may also directlyadjust rules.

Referring to FIG. 4B, the sequence 400 may also include operation 418,which comprises adding at least one new behavioral operator based on thefeedback received regarding the outputted results.

The sequence 400 may also include operation 420, which comprisesanalyzing the data. The data may be analyzed continuously, orrepeatedly, or only initially. The operation of analyzing the data mayutilize artificial intelligence methods that incorporatemachine-learning techniques to adapt the profiling system's 100operational rules, inference structure, and/or anomaly detectionperformance over time. Such methods may include reinforcement learning,where feedback is given when truth is determined regarding specifieditem(s) and/or individual(s), which permits correcting incorrectprofiles and reinforcing correct profiles. These methods may alsoinclude evolutionary computing in which alternative hypothesized rulesregarding the methods of generating answers to profiling queries can beoptimized over time through an iterative process of variation andselection. The profiling system 100 may incorporate other methods, suchas neural networks, decision trees, finite state machines, and/or otherfunctions, which serve to enhance the appropriateness, accuracy, andprecision of the profiling system 100 in response to user 302 inquiries.The results of checking the integrity of the data, the behavioraloperators, and detecting anomalies can be used to improve the machinelearning procedures. Furthermore, the results from the machine learningcan improve the behavioral operators and anomaly detection performance.In addition to the machine learning techniques, a user 302 may updatethe data sources, and profiling modules.

The profiling system 100 may use mathematical logic that is capable ofhandling linguistic concepts. One such logic that can accomplish this isfuzzy logic, which is a form of approximate reasoning. Fuzzy logicaccommodates approximate relationships, wherein data can be categorizedlinguistically (for example as high or low, or heavy or light), ratherthan numerically. Other methods of approximate reasoning could beincorporated in the profiling system 100. In this manner, the profilingsystem 100 can handle inquiries even if data are omitted, either insmall part, in large part, or completely, and/or if the data are deemedto have less than one-hundred percent reliability.

Two principal approaches to machine learning are called supervisedlearning and unsupervised learning. Machine learning may also includenormal statistical methods. With supervised learning, examples ofresults (outcomes) are available, and mathematical models are generatedto relate data inputs to results. The mathematical models may includebehavioral rules, and may also include mathematical constructions otherthan behavioral rules, such as a neural network, which can be used as anew part of the behavioral rule set. The mathematical models may be usedto produce rule-based scoring. As an example, a neural network mayassign a value to a particular set of inputs, for example, thearrangement of playing pieces on a checker board. Models may alsoinclude human expert rules, such as if A then B, and if C then D. In thecase of neural networks, evolutionary computing can be used to assignweights to inputs and nodes in the network, which may also be adjustedby gradient methods, annealing, and other meta-heuristics. Themathematical models can then be used to predict results, if new inputdata are inputted into the models. As an example, several featuresregarding a patient and the patient's mammogram may be inputted to aneural network to obtain an output regarding the level of risk thatcancer is present. As another example, feature data regarding moviesthat a person liked and did not like can be used generate a model topredict which movies the person will like and will not like.

The operation 420 of analyzing the data may comprise developingmathematical models to explain outcomes, which is discussed above. Thesequence 400 may also include operation 422, which comprises using themathematical models to generate at least one new behavioral operator(new behavioral rules or new mathematical models). The behavioraloperators do not have to be explicit behavioral rules, and for example,could be the output of a neural network (which may provide a correctoutput using logic that is not explicitly understood). The sequence 400may also include operation 424, which comprises including the at leastone new behavioral operator in the behavioral operators. The sequence400 may also include operation 426, which comprises using themathematical models to delete at least one behavioral operator. Thesequence 400 may also include operation 428, which comprises using themathematical models to modify at least one behavioral operator.

The operation 420 of analyzing the data may comprise detecting if thereare one or more anomalies in the data, instead of, or in addition todeveloping mathematical models to explain outcomes. The operation ofdetecting if there are any anomalies in the data, may also includeidentifying the anomalies. An anomaly is data that does not fit into anycluster. In specific applications, anomalies may be defined, forexample, as machine defects, or as the presence of factors that mayindicate terrorist activity. For example, hazardous materials (HAZMAT),may present an anomaly in shipping data that may indicate terroristactivity, and warrant further investigation. In other instances it maynot be known what types of anomalies may be present, and an anomaly maybe defined as anything unusual in the data.

Unsupervised learning will generally be used for detecting anomalies.Unsupervised learning concerns looking for patterns in data whenexamples of outcomes are not known. With unsupervised learning, examplesof outcomes are not available or are not used, the data may not belabeled, and the computer looks for patterns in the data, and may groupthe data into clusters. Clusters are formed so as to maximize similarityof data within each cluster, and to maximize differences betweendifferent clusters. Anomalies are an indication of unusualness, and maybe identified by mining data to find data that does not belong in acluster. Evolutionary computing, as well as other methods such ask-means and annealing, may be used for forming the clusters. Differentmodels may be generated for the data, for example, linear, or nonlinearmodels. It is possible to generate multiple reasonable sets of clustersfor the same data when conditioning on different aspects of the data'sproperties. For example, a penny, nickel, dime, and quarter (U.S. coins)could be clustered by assigning the penny and nickel to one cluster andthe dime and quarter to another, with the rule being that the clustersseparate the data based on unit value. An alternative would be to assignthe penny and dime to one cluster and the nickel and quarter to another,thereby partitioning based on the size of the coin. In general, theremay be more than two clusters, and determining an optimal number ofclusters is a problem of significant mathematical interest with a longhistory of work. Once the data are clustered, additional analysis canindicate which data least belong to any cluster, and are thereforeanomalous.

Mathematical statistical inference, time series analysis methods,pattern recognition, and/or evolutionary computation may be utilized toassess the normalcy of an event or condition. For example, an event orcondition may be described in conditional statements, such as, “If thereis a pattern of transporting bananas by truck across an internationalborder every Tuesday, and if it is Tuesday, and if the currenttransported item is not bananas, then increase the perceived riskassociated with the transported item.” Generally, anomaly detectionoperates by creating statistical descriptions and/or models of what isroutine behavior, and then identifies behaviors that are not routine.

The sequence 400 may also include operation 430, which comprisesperforming additional data integrity testing on a detected anomaly. Thesequence 400 may also include operation 432, which comprises generatingan alert concerning the detected anomaly, to notify the user 302 of theanomaly.

Referring to FIG. 4C, the sequence 400 may also include operation 434,which comprises altering at least one behavioral operator based on thedetected anomaly.

The degree to which a record is anomalous, or an outlier, can beclassified in linguistic categories such as “not anomalous,” “low”degree of anomaly, “medium” degree of anomaly, and “high” degree ofanomaly. These categories may be defined numerically, arbitrarily, foreach cluster. Following the evolution or other determination ofclusters, each of the records may have its rule-based risk scoreincremented according to its rank among the records. The amount a scoreis incremented may be as follows:

-   -   “not anomalous”: no incremental change of score;    -   “low” degree of anomaly: increment score by 1/60 (0.0167);    -   “medium” degree of anomaly: increment score by 2/60 (0.0333);        and    -   “high” degree of anomaly: increment score by 3/60 (0.05).

The sequence 400 may also include operation 436, which comprisesreceiving feedback concerning system performance. The sequence 400 mayalso include operation 438, which comprises adjusting at least oneparameter based on the feedback received concerning system performance.For example, in a shipping screening implementation, the user 302 mayinput feedback information to the profiling system 100 to inform theprofiling system 100 that too many alerts are being generated, orconversely, that not enough alerts are being generated. As anotherexample, the profiling system 100 may require more time than isavailable for a particular application, and consequently, the parametersof the algorithm are adjusted to make the algorithm converge faster. Asanother example, if an evolutionary algorithm is used for clusteringdata, based on clustering performance, the user 302 may desire to revisethe number of clusters from, for example, two to three or more, or fromfour to three or less. Other adjustments might be made to the parametersof the clustering algorithm, including the population size, the numberof parents, the number of offspring per parent, the types of variationoperators, the type of selection operator, and so forth. As anotherexample, in the shipping container screening embodiment, based onevidence obtained by opening a container and examining the contents, thecorrectness of the possible classification of the contents by thealgorithm can be fed back to the method to have the method adjust theparameters of its functions. If the process utilizes a neural network,the error (if any) of the classification could be used as a basis foradjusting the weights (parameters) of the neural network to compensatefor the error. As another example, the user's 302 goals could change,and the user 302 could provide feedback to the profiling system 100 sothat results will be generated consistent with the new goals.

The sequence 400 may also include operation 440, which comprisesproactively generating at least one suggestion. Suggestions may concernproperties of available data that are of potential interest to the user.Suggestions may be generated continuously, or repeatedly, or only once.The sequence 400 may also include operation 442, which comprisesoutputting at least one of the generated suggestions. The sequence 400may also include operation 444, which comprises soliciting feedbackconcerning the at least one generated suggestion. The sequence 400 mayalso include operation 446, which comprises receiving feedback (from theuser 302) concerning at least one of the at least one generatedsuggestions. The sequence 400 may also include operation 448, whichcomprises interpreting the feedback (or lack of feedback) receivedconcerning at least one of the at least one generated suggestions. Theprofiling system 100 may analyze the data continuously, or repeatedly,or only once, in order to generate suggestions. As an example, theprofiling system 100 may proactively suggest to the user 302 that theuser 302 may be interested in dating a person in a dating service database, who the profiling system 100 has determined may be of interest tothe user 302. The profiling system 100 outputs the suggestion so that itcan be presented to the user 302 (for example on a display 114), andalso asks the user 302 whether the user 302 is interested in thesuggested person. The profiling system 100 may adjust its behaviorconcerning future suggestions, based on the feedback (or lack offeedback) received from the user 302 regarding the person suggested bythe profiling system 100. Thus, the profiling system 100 isintelligently interactive, and learns to offer data relevant to the user302 through the class of machine learning called reinforcement learning.

The sequence 400 need not end after operation 448. Generally, theoperations of the sequence 400 may be repeated as many times as desired,and as long as desired. For example, the operation 404 of receiving datamay be repeatedly performed. Generally one, several, or all, of theoperations may be repeated. Results may be improved with each iterationof the sequence 400.

The decisions, factual instantiations, and other effects of theprofiling system 100 may be stored in a data warehouse, for use insubsequent profiling operations, thereby allowing the profiling system100 to build on its decision making and prior performance. Further,additional databases could be added to the system in a modular fashionfor enhanced performance.

In summary, some examples of the invention relate to a method forprofiling multiple items and/or individuals, and outputting results,which for example, could indicate a perceived level of risk, in anintuitive user-friendly manner. An exemplary embodiment is a profilingsystem for categorizing risk, which includes a primary process ofrule-based risk scores, and a secondary procedure for anomaly detectionusing evolutionary computation. The profiling 100 system supplementshuman expertise with machine learning and data mining tools, such asevolutionary computing. Many of the examples of the invention benefitfrom the synergy of performing evolutionary computing for performingdata modeling and continuous anomaly detection, in combination withproactively generating suggestions and soliciting feedback from a userfor adjusting the system.

III. Other Embodiments

The preceding disclosure describes a number of illustrative embodimentsof the invention. It will be apparent to persons skilled in the art thatvarious changes and modifications can be made to the describedembodiments without departing from the scope of the invention as definedby the following claims. Also, although elements of the invention may bedescribed or claimed herein in the singular, the plural is contemplatedunless limitation to the singular is explicitly stated.

1. A signal bearing medium tangibly embodying a program ofmachine-readable instructions executable by a digital processingapparatus to perform a method for identifying at least one property ofdata, the method comprising the following operations: receiving data;making assessments regarding the data; applying at least one behavioraloperator; analyzing the data, wherein the operation of analyzing thedata comprises detecting if there are any anomalies in the data;outputting results; receiving feedback concerning system performance;and adjusting at least one parameter based on the feedback receivedconcerning system performance, wherein the at least one parameter is aparameter of a machine learning method.
 2. The signal bearing medium ofclaim 1, wherein the operations further comprise repeating the receivingdata, making assessments, applying, analyzing, outputting, receivingfeedback, and adjusting operations.
 3. The signal bearing medium ofclaim 1, wherein the machine learning method involves a neural network,and wherein the at least one parameter is a weight.
 4. The signalbearing medium of claim 1, wherein the machine learning method is anevolutionary algorithm.
 5. The signal bearing medium of claim 1, whereinthe machine learning method is an evolutionary clustering algorithm. 6.The signal bearing medium of claim 1, wherein the machine learningmethod is reinforcement learning.
 7. The signal bearing medium of claim1, wherein the machine learning method is hill-climbing.
 8. The signalbearing medium of claim 1, wherein the machine learning method isannealing.
 9. The signal bearing medium of claim 1, wherein the machinelearning method is meta-heuristics.
 10. The signal bearing medium ofclaim 1, wherein the operation of making assessments regarding the datafurther comprises making assessments regarding features.
 11. The signalbearing medium of claim 1, wherein the operations further comprisereceiving user knowledge.
 12. The signal bearing medium of claim 1,wherein the operation of analyzing the data further comprises repeatedlyanalyzing the data.
 13. The signal bearing medium of claim 1, whereinthe operation of analyzing the data further comprises developing atleast one mathematical model to explain outcomes.
 14. The signal bearingmedium of claim 13, wherein the operations further comprise: using theat least one mathematical model to generate at least one new rule; andusing the at least one new rule as one of the behavioral operators. 15.The signal bearing medium of claim 13, wherein the operations furthercomprise using the at least one mathematical model to delete at leastone behavioral operator.
 16. The signal bearing medium of claim 13,wherein the operations further comprise using the at least onemathematical model to modify at least one behavioral rule.
 17. Thesignal bearing medium of claim 1, wherein the operations furthercomprise performing data integrity testing on a detected anomaly. 18.The signal bearing medium of claim 1, wherein the operations furthercomprise generating an alert concerning a detected anomaly.
 19. Thesignal bearing medium of claim 1, wherein the operations furthercomprise altering at least one operational rule based on a detectedanomaly.
 20. The signal bearing medium of claim 1, wherein theoperations further comprise: proactively generating at least onesuggestion; outputting the at least one generated suggestion; andsoliciting feedback concerning the at least one generated suggestion.21. The signal bearing medium of claim 20, wherein the operationsfurther comprise: receiving feedback concerning at least one of the atleast one generated suggestions; and interpreting the feedback receivedconcerning at least one of the at least one generated suggestions. 22.The signal bearing medium of claim 20: wherein the operation ofproactively generating at least one suggestion comprises repeatedlygenerating suggestions; and wherein the operation of outputting the atleast one suggestion comprises outputting each of the generatedsuggestions.
 23. The signal bearing medium of claim 1, wherein theoperation of receiving data comprises repeatedly receiving data.
 24. Thesignal bearing medium of claim 1, wherein the data comprises commercialdata.
 25. The signal bearing medium of claim 1, wherein the datacomprises government data.
 26. The signal bearing medium of claim 1,wherein the operations further comprise: receiving feedback regardingthe outputted results; and adding at least one new operational rulebased on the feedback regarding the outputted results.
 27. The signalbearing medium of claim 1, wherein the operations further comprise:receiving feedback regarding the outputted results; and adjusting atleast one operational operator based on the feedback received regardingthe outputted results.
 28. The signal bearing medium of claim 1, whereinthe operation of outputting results comprises: outputting rules andresults; and outputting information configured to display the rules andresults according to user preferences.
 29. The signal bearing medium ofclaim 1, wherein the operation of outputting results comprisesoutputting information configured to indicate membership in at least onemembership function in a plurality of membership functions.
 30. Thesignal bearing medium of claim 1, wherein the operation of outputtingresults comprises outputting information configured to display aplurality of membership functions and an indicator showing arelationship between the results and the membership functions
 31. Thesignal bearing medium of claim 30, wherein each membership function inthe plurality of membership functions is associated with a respectivelevel of risk.
 32. The signal bearing medium of claim 1, wherein the atleast one property of the data comprises the risk presented by ashipment.
 33. The signal bearing medium of claim 1, wherein the at leastone property of the data comprises the risk presented by a shippingcontainer.
 34. The signal bearing medium of claim 1, wherein the atleast one property of the data comprises the risk that an individual isa terrorist.
 35. The signal bearing medium of claim 1, wherein the atleast one property of the data comprises the risk associated with acredit transaction.
 36. The signal bearing medium of claim 1, whereinthe at least one property of the data comprises the risk that cancer ispresent.
 37. The signal bearing medium of claim 1, wherein the at leastone property of the data comprises the chances that a person will enjoydating another person.
 38. The signal bearing medium of claim 1, whereinthe at least one property of the data comprises the chances that aperson will enjoy a particular movie.
 39. A signal bearing mediumtangibly embodying a program of machine-readable instructions executableby a digital processing apparatus to perform a method for identifying atleast one property of data, the method comprising the followingoperations: receiving data; making assessments regarding the data;applying at least one behavioral operator; outputting results; receivingfeedback regarding the outputted results; adjusting at least onebehavioral operator based on the feedback received regarding theoutputted results; and analyzing the data, wherein the operation ofanalyzing the data comprises generating at least one machine generatedmathematical model to explain outcomes.
 40. The signal bearing medium ofclaim 39, wherein the operations further comprise: proactivelygenerating at least one suggestion; outputting the at least onegenerated suggestion; and soliciting feedback concerning the at leastone generated suggestion.
 41. The signal bearing medium of claim 40,wherein the operations further comprise: receiving feedback concerningat least one of the at least one generated suggestions; and interpretingthe feedback received concerning at least one of the at least onegenerated suggestions.
 42. The signal bearing medium of claim 40,wherein the operations further comprise: receiving feedback concerningsystem performance; and adjusting at least one parameter based on thefeedback received concerning system performance, wherein the at leastone parameter is a parameter of a machine learning method.
 43. Thesignal bearing medium of claim 39, wherein the operation of analyzingthe data comprises detecting if there are any anomalies in the data. 44.A signal bearing medium tangibly embodying a program of machine-readableinstructions executable by a digital processing apparatus to perform amethod for identifying at least one property of data, the methodcomprising the following operations: receiving data; making assessmentsregarding the data; checking integrity of the data; applying at leastone behavioral operator; using machine learning to detect if there areany anomalies in the data; outputting results; proactively generating atleast one suggestion; outputting the at least one generated suggestion;and soliciting feedback concerning the at least one generatedsuggestion.
 45. The signal bearing medium of claim 44, wherein theoperations further comprise repeating the receiving data, makingassessments, applying, outputting, receiving feedback, and adjustingoperations.
 46. The signal bearing medium of claim 44, wherein theoperations further comprise: receiving feedback concerning at least oneof the at least one generated suggestions; and interpreting the feedbackreceived concerning at least one of the at least one generatedsuggestions.
 47. The signal bearing medium of claim 44, wherein theoperations further comprise: receiving feedback concerning systemperformance; and adjusting at least one parameter based on the feedbackreceived concerning system performance, wherein the at least oneparameter is a parameter of a machine learning method.
 48. The signalbearing medium of claim 44, wherein the operation of using machinelearning to detect if there are any anomalies in the data comprisesusing evolutionary learning.
 49. The signal bearing medium of claim 44,wherein the operations further comprise analyzing the data, and whereinthe operation of analyzing the data comprises generating at least onemachine generated mathematical model to explain outcomes.
 50. A signalbearing medium tangibly embodying a program of machine-readableinstructions executable by a digital processing apparatus to perform amethod for identifying at least one property of data, the methodcomprising the following operations: receiving data; making assessmentsregarding features and the data; receiving user knowledge; applying atleast one behavioral operator; outputting results; wherein the operationof outputting results comprises outputting information configured todisplay a plurality of membership functions and an indicator showing arelationship between the results and the membership functions; receivingfeedback regarding the outputted results; adjusting at least one of theat least one behavioral operators based on the feedback receivedregarding the outputted results; adding at least one new behavioraloperator based on the feedback received regarding the outputted results;analyzing the data; wherein the operation of analyzing the datacomprises developing at least one mathematical model to explainoutcomes; using the at least one mathematical model to generate at leastone new behavioral operator; including the at least one new behavioraloperator in the behavioral operators; using the at least onemathematical model to delete at least one behavioral operator; using theat least one mathematical model to modify at least one behavioraloperator; wherein the operation of analyzing the data further comprisesdetecting if there are any anomalies in the data; performing additionaldata integrity testing on a detected anomaly; generating an alertconcerning the detected anomaly; altering at least one behavioraloperator based on the detected anomaly; receiving feedback concerningsystem performance; adjusting at least one parameter based on thefeedback received concerning system performance, wherein the at leastone parameter is a parameter of a machine learning method; proactivelygenerating at least one suggestion; outputting the at least onegenerated suggestion; soliciting feedback concerning the at least onegenerated suggestion; receiving feedback concerning at least one of theat least one generated suggestions; and interpreting the feedbackreceived concerning at least one of the at least one generatedsuggestions.
 51. A signal bearing medium tangibly embodyingmachine-readable code executable by a digital processing apparatus foridentifying at least one property of data, the code comprising: a dataintegrity module configured to examine integrity of the data; abehavioral operator module configured generate and evaluate behavioraloperators; an anomaly detection module configured to detect anomalies inthe data; a machine learning module configured to analyze the data; andan interface/controller module coupled to the data integrity module, thebehavioral operators module, the anomaly detection module, and themachine learning module; wherein the interface/controller module isconfigured to receive the data.
 52. The signal bearing medium of claim51, wherein the interface/controller module is further configured to:proactively generate suggestions; output the generated suggestions; andsolicit feedback concerning the generated suggestions.
 53. The signalbearing medium of claim 52, wherein the interface/controller module isfurther configured to interpret feedback concerning the generatedsuggestions.
 54. The signal bearing medium of claim 51, wherein theinterface/controller module is further configured to: receive feedbackconcerning system performance; and adjust parameters based on thefeedback received concerning system performance.
 55. The signal bearingmedium of claim 51: wherein the interface/controller module is furtherconfigured to output results and to receive feedback regarding theoutputted results; and wherein the behavioral operators module isfurther configured to adjust the behavioral operators based on thefeedback received regarding the outputted results.
 56. The signalbearing medium of claim 51: wherein the interface/controller module isfurther configured output results and to receive feedback regardingoutputted results; and wherein the behavioral operators module isfurther configured to add new behavioral operators based on the feedbackreceived regarding the outputted results.
 57. A computer data signalembodied in a carrier wave embodying a program of machine-readableinstructions executable by a digital processing apparatus to perform amethod for identifying at least one property of data, wherein the methodcomprises the following operations: receiving data; making assessmentsregarding the data; applying at least one behavioral operator; detectingif there are any anomalies in the data; outputting results; receivingfeedback concerning system performance; and adjusting at least oneparameter based on the feedback received concerning system performance,wherein the at least one parameter is a parameter of a machine learningmethod.
 58. A profiling system, comprising: a storage; and a processorcoupled to the storage, wherein the processor is programmed to performthe following operations: receiving data; making assessments regardingthe data; applying at least one behavioral operator; outputting results;receiving feedback regarding the outputted results; adjusting at leastone behavioral operator based on the feedback received regarding theoutputted results; and analyzing the data, wherein the operation ofanalyzing the data comprises generating at least one machine generatedmathematical model to explain outcomes.
 59. The profiling system ofclaim 58, wherein the operations further comprise: proactivelygenerating at least one suggestion; outputting the at least onegenerated suggestion; and soliciting feedback concerning the at leastone generated suggestion.
 60. A profiling system, comprising: means forreceiving data; means for making assessments regarding the data; meansfor applying at least one behavioral operator; means for outputtingresults; means for receiving feedback concerning system performance;means for adjusting at least one parameter based on the feedbackreceived concerning system performance, wherein the at least oneparameter is a parameter of a machine learning method; means foranalyzing the data; means for proactively generating at least onesuggestion; means for outputting the at least one generated suggestion;and means for soliciting feedback concerning the at least one generatedsuggestion.
 61. A method for identifying at least one property of data,the method comprising the following operations: receiving data; makingassessments regarding the data; applying at least one behavioraloperator; analyzing the data, wherein the operation of analyzing thedata comprises detecting if there are any anomalies in the data;outputting results; receiving feedback concerning system performance;and adjusting at least one parameter based on the feedback receivedconcerning system performance, wherein the at least one parameter is aparameter of a machine learning method.
 62. A method for identifying atleast one property of data, the method comprising the followingoperations: receiving data; making assessments regarding the data;applying at least one behavioral operator; outputting results; receivingfeedback regarding the outputted results; adjusting at least onebehavioral operator based on the feedback received regarding theoutputted results; and analyzing the data, wherein the operation ofanalyzing the data comprises generating at least one machine generatedmathematical model to explain outcomes.
 63. A method for identifying atleast one property of data, the method comprising the followingoperations: receiving data; making assessments regarding the data;checking integrity of the data; applying at least one behavioraloperator; generating at least one machine generated mathematical modelto explain outcomes; outputting results; proactively generating at leastone suggestion; outputting the at least one generated suggestion; andsoliciting feedback concerning the at least one generated suggestion.